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ABSTRACT 

The purpose of this paper is to outline elements that 
are critical for an effective analysis and revision of instructional 
units of a learning system. No attempt made to present an 
exhaustive list of critical elements nor a list of elements that are 
necessarily mutually exclusive. The elements that are presented in 
the paper, although consisting of a first approximation, appear to 
subsume characteristics that account for most of the ineffectiveness 
of instructional systems. They are the distillation of extensive 
experience in the analysis and revision of an actual individualized, 
performance- based instructional system that constituted the entire 
teacher education program of a School of Education. Examples to 
illustrate the categories will be drawn from actual practice. 

(Author) 



£00 58287 



U.8. DEPARTMENT OF HEALTH, 

EDUCATION ft WELFARE / 

OFFICE OF EDUCATION 
THIS OOCUMENT HAS BEEN REPRO- 
OUCEO EXACTLY AS RECEIVEO FROM 
THE PERSON OR ORGANIZATION ORIG- 
INATING IT. POINTS OF VIEW OR OPIN- 
IONS STATEO 00 NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU- 
CATION POSITION OR POLICY. 



1 

CRITICAL ELEMENTS IN ANALYSIS AND REVISION OF INSTRUCTIONAL SYSTEMS 



John R. Pyper 
Weber State College^ 



CQ 

o 

o 

pH 

o 

© 

S3 

&-< 



As increasing concern is being manifest by teachers, students, and 
taxpayers for instruction that is more relevant, interesting, effective, and 
efficient, the relative importance of the various roles of the classroom 
instructor are shifting significantly. This shift in the roles of the classroom 
instructor is from that of largely a dispenser of information toward that of 
more of a designer and manager of instruction. Recently, increasing attention 
is being given to the need fcr more emphasis of the latter role (e.g. Merrill, 
1971, 1968; and Keller, 1968). Increasing interest is also being expressed 
in the need fora better methodology of curriculum evaluation (e.g. Weiss, 
1971). An excellent initial contribution in this area is the recent Handbook 
of Procedures for the Design of Instruction by Briggs (1970) although it does 
not attempt to deal at length with the somewhat different tasks in revising 
instruction. In addition, there is an increasing wealth of information on 
instructional design, much of which is directly relevant to the problems of 
the formative evaluation of curriculum developments. 

This paper is concerned with some of the "growing pains" of those 
instructors who are striving to effect such a transformation in themselves. 

In particular, it is concerned with those, who having started to revise their 
bailiwick of instruction, find that their first draft efforts have not resulted 
in as much progress as they had anticipated and who even may be sorely 
tempted to conclude that they are even farther from their goal than when 
they started. 

The problem that arises for these adventuresome individuals (who having 
departed from the way of their complacent colleagues, but not having attained 
that blessed state to which they aspire - the well-tempered instructional 
program - yet still with enough energy, faith, and enthusiasm to keep going) 
is in the form of questions such as: Where do I go from here? What needs 
revising? What kind of help could I utilize? What should I start revising 
first? Perhaps some of these questions cannot be answered satisfactorily, 
but the fact is that they are being asked, and some useful observations can 
be made to those teachers who are grappling with these problems for the 
first time. 



^Paper prepared for presentation at California Educational Research 
Association Annual Meeting . San Diego. April 29-30, 1971. 
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As noted, there is no lack of information on instructional design, although 
much of it is not popularized or disseminated as well as it could be. 
Likewise, there is no dearth of admonitions by experienced developers that 
one should not expect that his first draft version will be satisfactory. Unfor- 
tunately, it does not follow that there is adequate information available 
regarding heuristics and strategies for revising first draft efforts. 

It is true that there have been some significant contributions concerning 
this topic in recent years, among which the work by Scriven (1967) has been a 
particularly appealing example. It is also true that there is, scattered 
among textbooks on curriculum, some attention to the problems of revision. 

But these contributions are focused largely on the comprehensive problems 
of the manager of large-scale curriculum development projects, rather than 
on those of a classroom instructor trying to revise a course or a smaller 
unit of instruction. It is to the concerns of the latter group that this paper 
is particularly directed. 

Much of what follows in this paper is the result of experience of the 
author for an academic year with the faculty df a school of education dedi- 
cated and committed to the revision of their individual instructional units, 
as well as the overall instructional system of the school of education. 

Many of the examples used in this paper are drawn from the pilot version of 
their instructional units. This paper grew out of a concern to define and 
describe the major tasks that they appear to be facing as they revise their 
first draft efforts. At this point it is critical to note that before modification 
of their instructional units and system was initiated, a model of the basic 
instructional unit was developed. This model contains the fundamental 
components of an instructional system and therefore a description of its 
characteristics provides the first approximation of a guide, not only for the 
development of instruction, but also for the analysis and revision of first 
draft efforts. The components are as follows: 

1. Public behavioral objectives (i.e. they are stated to the student) 
in all relevant domains of learning and future job tasks (cognitive, 
affective, motor-skill and social-interactive) with a distinct 
emphasis upon performance objectives. 

2. Criterion- referenced and behavioral objective-referenced testing 
procedures which require students to demonstrate minimal levels 
of proficiency on the objectives of a unit of instruction before 
exiting . 

3. Unit pre-testing permitting student competencies to be evaluated to 
determine which objectives a student may already have accomplished 
andthereby allow’ him to bypass the related learning requirements. 
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4. Flexible scheduling, permitting students to progress at a pace they 
set, rather than at an arbitrary rate. 

5. Development of learning experiences that are as relevant as reason- 
ably possible to the attainment of the objectives of a unit of instruc- 
tion (e.g. observation, practice, experiences, etc., which dramat- 
ically drops the amount of passive learning via lectures, etc.). 

6. Self-contained units of instruction that are intrinsically cohesive 
rather than survey type instruction. 

7. Alternative objectives (and accompanying instructional units) 
from which the student can select to complete requirements in a 
particular area. 

8. Shifting faculty tasks more to the roles of designers and managers 
of instruction from that of dispensers of instruction (e.g. reduction 
in lecturing) . 

9. Development of differentiated staffing so that procedural activities 
such as testing, video-taping, etc., are not the direct responsibility 
of the instructor. 

The above list identifies and makes somewhat explicit the nature of 
several fundamental components of instruction, namely objectives , evaluation , 
learning activitie s, management , and organization of learning activities 
(e.g. according to task anal ysis of hierarchical structure) . Two other 
components that have not been mentioned that should be included are pre- 
requisites (entering abilities of students) and theoretical rationale (philoso- 
phy, learning theory, etc.). At this point, we have a list of components 
of instruction which as designer of instruction could use as a guide in 
evaluating his efforts as well as a brief description of criterion levels of 
application for most of them. 

Some guides for development and revision of instruction tend to peter 
out after listing similar basic components of instruction with perhaps criterion 
levels of application in some categories (e.g. Cunningham, 1971; Briggs, 

1970). In addition, there has been extensive, although compartmentalized, 
development of development and revision strategies with regard to some of 
these components, particularly testing, objectives (behavioral variety) , 
and various specific types of learning and learning activities. Compartmen- 
talized application and thinking about these methodologies has inhibited the 
development of a comprehensive, general framework in which strategies and 
procedures for development and revision of all of the components of the 
instructional unit or system can be organized. 
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Presentation of the initial draft of such a framework is the major objective 
of this paper. Figure 1 presents this framework in a two-dimensional 
matrix form. The elements of the Components dimension have been briefly 
described above. The remainder of the paper will be concerned with an 
examination of the Attributes dimension. It consists of categories that 
have been found useful in analysis and revision activities this past year 
of the teacher education instructional system described above. In addition, 
the attributes include categories used by various instructional and test 
designers and developers in making value judgments (e.g. Briggs, 1970; 

Scriven, 1967; Weiss, 1971). 

This paper is too brief to be able to consider at length what the individual 
cells of this matrix might represent or contain. Nor can evaluation procedures 
pertinent to the various areas of the matrix be discussed in detail. Such 
details will have to be covered in a later, more extensive treatment . In 
addition, there is yet unexplored and undeveloped territory in this matrix.. 

The primary purpose of this paper is to present this matrix as a possible 
framework by which increasing information about analysis and revision of 
instructional systems can be organized and as a guide in developing analysis 
and revision procedures. Since the principal objective in developing this frame- 
work is to develop effective instructional analysis and revision procedures, 
the attributes will be discussed in terms of the type of analysis that would 
be done . 

Relevance Analysis ; This type of analysis is concerned with determining 
the significance (practical, social, job, etc.) of the various components of 
the instructional unit. Relevance is a loaded term in these times and the 
temptation was strong to substitute a more innocuous term such as signifi- 
cance . But in spite of some misuses of the term, it is sufficiently repre- 
sentative of a valid and sincere concern regarding instruction. Questions 
such as the following are becoming of such great importance to more and 
more students, taxpayers, educators and pressure groups that often they have 
to be dealt with first by the innovator: 

1. Are the objectives of your instruction the most appropriate (relevant) 
ones in preparing the students to be effective in the immediate and 
long-term future. Are they meeting their needs? 

2. Are the learning experiences the most relevant ones for the students, 
given their backgrounds, and the instructional objectives? 

3 . Are the testing procedures the most relevant that can be developed 
as a manifestation of the objectives? 

Likewise, questions can be asked about the other components. 
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Figure 1 Components of instructional systems and attributes of these components. 
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If one does not have immediate worries about the above mentioned 
pressures, there is another with which he may become involved. If he 
wishes to develop a product that may be accepted by others and even 
marketable economically, he must be concerned with the relevance of what 
he is producing. In a phrase, he must be up-to-date. It is true that which 
is of enduring value must be distinguished from that which is of fleeting 
fashionability, but one must seek to avoid revising and improving material 
that is obsolete. 

One application of the use of this category will be illustrated by an 
example from the area of testing. Typically an analysis of instructor- 
made tests of college instructors will include sorting the items into cate- 
gories approximating the Cognitive Taxonomy of Bloom et al. Such an 
example is presented by Ebel (1966) in which tests are analyzed for the 
percentage of items in the following categories: Content details, Vocabu- 
lary, Facts, Generalizations, Understandings, Applications. Having 
a specification of the objectives of an instructional unit, one can go a 
step further and also classify questions as relevant or irrelevant in each 
of the above mentioned categories. Some examples of i rrelevant test 
items are as follows: 

1. Recall of the label of a techniques procedure or criteria that is 
tangentially related to the objectives of a unit of instruction. 

Example: A question from the proficiency assessment of a 
unit of instruction on Self-Concept: 4 

If a self-report instrument is measuring what it claims to measure, 
it is considered: 

a. accurate 

b. valid 

c. reliable 

d. consistent 

2. Recall or application of the characteristics of structure of a dis- 
cipline (eig. English, Social Sciences, etc.) at the expense of 
recall or application of the knowledge of research findings or 
pedagogic techniques for introducing students at various ages to 
appropriate aspects of the discipline. 

Example: List the characteristics that define and distinguish the 
following areas of social science and give examples to illustrate 
the characteristics. 

4 

: This question was anexcellent discriminator although’ a bit hard.' --4'3% 
chose b, the correct answer, and 57% chose c. The discrimination Correlation 
of alternative b was .56 and alternative c was -.56. Therefore, it represents 
an item that would probably remain in most tests . It found its way into this 
test because the information was briefly referred to in one of the reading 
O assignments. n 
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3. Recall or application of facts that are relevant to the discipline 
but which are not relevant to the area of overlap between that 
discipline and teacher education. 

Example: Questions concerning developmental characteristics of 
children ages 1-5 in a unit on Growth and Development for Elemen- 
tary Teachers. 

Overlap Analysis: This analysis would be concerned with ascertaining the 
degree of agreement of the various components of the instructional unit 
(or system) . This term was chosen because of its descriptiveness. For 
example, what is the degree of agreement between the stated objectives and 
the manifest objectives that constitute the evaluation procedures ? What 
is the degree of agreement among both of the above and the content of the 
learning experiences? What is the degree of agreement among the pre-test, 
any self-test, and the post test (proficiency assessment) . Are alternate 
forms of the various tests reasonably equivalent? 

Scriven (1967, p. 57) has referred to the analysis of such cohesiveness 
as consistency analysis. To some degree it is similar to relevance analysis. 
But even though these categories are not mutually exclusive it is included 
because it is felt that it will have a practical influence on the extensiveness 
of ones' analysis activities. Appendix A contains an example of the analysis 
of the degree of overlap between (1) the stated objectives that appear in the 
beginning of the printed materials of this unit of instruction and (2) the 
manifest objectives represented by the tasks of the self-test and proficiency 
assessment. 

It will be noted, in this particular example, which is a first draft effort, 
that six of the nine types of tasks on the tests do not overlap and that two 
of the nine tasks are not represented by a stated objective. 

Transfer Analysis : Another type of "overlap" is that of the content of the 
instructional and evaluational tasks of an instructional unit with the content 
of the day-to-day activities of the student after graduation. In the case of 
the instructional system the author has been working with, it would be teaching 
tasks in the classroom of the future teacher. A basic question to be asked is: 
What percentage of the criterion level of performance of the tasks of the 
proficiency assessment will be utilized, what percentage of time, in the 
applied day-to-day tasks of teaching? 

This type of analysis can be subdivided into more detailed questions 
such as, What percentage of criterion performance is relevant to the dom- 
inant tasks of the first year or so of teaching versus later years? Also, it 
can be subdivided into analysis of intermediate steps to full application 
such as, how much transfer occurs to intermediate activities such as peer- 
teaching, microteaching, student teaching, etc? 
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As may have been noticed by the use of relevance and overlap in pre- 
ceding sentences, this category has much in common with the previous two 
categories. But the term transfer has been purposely chosen to represent a 
specific type of relevance and overlap. Also, use of this term will facilitate 
the utilization of the wealth of research concerning transfer. 

Mo tivational Analysis : This deals with the assessment of the arousal value 
of the instructional components. Some, and sometimes much, of the success 
or failure of a set of instructional materials depends upon how the students 
view the materials. If the students are actively and diligently participating in 
the learning experiences, achieving criterion on the proficiency assessment, 
expending an appropriate amount of effort on the learning and testing exercises 
and have a positive attitude toward their learning experiences and accomplish- 
ments, then the instruction would seem rather successful (although it could 
possibly not be relevant). If one of these aspects is missing, or if another 
element is involved such as a high expenditure of time and energy on tangen- 
tial administrative procedures by the students, the intended outcomes can 
be inhibited from occuring. In order to determine if such is the case, a 
representative sample of the students must be observed and consulted. 

Typically, if student initiative and effort is not at a desired level, it is be- 
cause the student does not see the need for such effort. Furthermore, even 
though he may see the need but can "pass" with a minimal amount of effort, 
he will probably do so. For example, if students find out that they can 
achieve criterion on the proficiency assessment without engaging in some of 
the learning experiences, it could either mean that the proficiency assess- 
ment is inadequate, that the learning experiences are superfluous, or that 
the' objectives are too elementary for the student population. 

Some consider that asking students to express their opinions concerning 
the relevance of the objectives, learning experiences, and testing procedures, 
is a source of "irrelevant" information because students change their minds 
about their educational experiences after they get on "the firing line". Never- 
the less, since students cannot help but form attitudes about the curriculum, 
and since negative or even neutral attitudes will affect their initiative, it 
will be advantageous in the long run to adjust one or more elements in the 
curriculum when low motivation is detected. 

Adequacy Analysis : The adequacy of an instructional unit is closely related 
to the transfer characteristics of the unit. The purpose of this analysis 
is to determine the adequacy of the stated objectives and learning experiences 
to cause the student to achieve criterion on the proficiency assessment. 

It would also include analysis of the adequacy of the proficiency assessment 
in predicting on-the-job applications, such as teaching. The concepts of 
validity and reliability, as particularly developed in the area of test con- 
struction would be appropriate subsumed in this category. Indeed the gen- 
eral nature of the concepts could be utilized in designing procedures for 
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evaluating the objectives, learning experiences, and other appropriate 
components . 

i The adequacy of the components of an instructional unit may be 

over-adequate as well as inadequate. At times too much emphasis can be 
placed on a particular set of objectives (oftimes an instructor s pet objec- 
tives) resulting in a stringent criterion level that results in tasks more 
demanding than those that will occur in the typical classroom. One dis- 
advantage of this overemphasis, aside from those involving motivation, is 
that the prospective teachers tend to develop expectations of student per- 
i formance that are unreasonably high. In addition, because less time is 

i spent on the development of other abilities , they are inadequately prepared 

! to cope with other teaching problems . 

More often, however, the instructional unit tends to be inadequate 
rather than overadequate . This is largely due to the fact that the proficiency of 
| the student is determined for only some of the objectives that are necessary 

| for satisfactory implementation of various teaching behaviors. A common 

i example is the development and assessment of the knowledge of various 

| teaching activities (questioning techniques , contingency management proced- 

ures, etc.) without practice and application. Almost as bad is the expecta- 
tion that a few superficial exercises (some almost rote) are sufficient to 
develop skills which are complex and need to be spontaneous to be effec- 
tive. Finally, too often the affective area has been completely neglected 
and no attempt is made to determine, much less modify, the students' 
attitudes toward the knowledge and skills they should develop. 

A thorough analysis of the adequacy of an instructional unit will involve 
Transfer analysis. However, a preliminary analysis can occur by answering 
two questions: 

1. How many domains of objectives are involved in the teaching tasks 
with which this instructional unit is concerned? 

[ 2. Are all domains appropriately covered in the evaluational and 

instructional activities? 

Inadequacy may be occuring because relevant objectives from domains 
! such as the motor-skill, affective or social-interactive may have been over- 

looked. They may have been overlooked because some of the steps of the 
! four-step learning model were skipped. The four steps are: 

1. Presentation of information. 

2 . Demonstration . 

3. Controlled practice . 

4. Applied practice. 
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Neglect of or inadequate development of any one of the steps may 
result in the omission of one or more seemingly minor, yet critical, sub- 
objectives . 

Efficiency Analysis : The major purpose of this analysis is to determine 
how much superfluous effort the students may be expending . It may be well 
to pay attention to the efficiency of the instructor's activities as well. This 
category is closely related to the adequacy category. However, adequacy 
and efficiency are somewhat independent. The analysis of efficiency involves 
the determination of whether the student is receiving too much or too little 
information, too much or too little practice, or too many or too few testing 
tasks. In addition to quantity, the quality of the instruction can be too high 
as well as too low. Instruction can be too well organized. It can appear 
so easy that the student is not sufficiently motivated to practice diligently 
or doesn't gain experience with common problems and thus is inadequately 
prepared. As well as too much or too little of something, there can be the 
problem of the inappropriateness of the instruction or the evaluation. (That 
aspect should show up in the overlap analysis .) 

Examples in this area would include the following: 

1. The student may be required to practice a skill 200 times when 20 
may be adequate, or 2000 may be necessary. 

2. The student may be required to read 40 pages when the necessary 
information could be extracted from two pages . 

3. The student may be required to complete tasks that he will never do 
when he teaches. 

4. The student may be required to prepare materials that could have 
been made in quantity much more efficiently and sold to him at 
minimal cost. 

5. Information may be presented at a slower rate than that possible 
with other alternatives and in addition no record is available for 
the student to review. This is typical of the lecture method. 



Some Brief Comments Regarding Implementation 

The application of these categories of analysis can occur throughout 
the development of an instructional unit . If the unit is part of a constantly 
evolving and self-regulating instructional system, these analyses will con- 
tifiue’to some degree. At the beginning stages of development, the nature 
of the evaluation procedures should be less structured and less obtrusive 
than in later stages of development. Thus a third dimension is added to 
the model presented in Figure One, namely the developmental stage of the 
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instructional unit or system. Atkin (1968), for example, suggests that 
forcing rigorous articulation at an early date of a curriculum developer's 
objectives may unduly limit the development and contribution of the proj- 
ect by causing premature closure. Likewise, similar results could occur 
concerning other components and attributes> 

Various classes of individuals as well as techniques should be utilized in 
obtaining effective evaluations. Some evaluations can be done by the 
author. Others will require the use of colleagues and peers, students, 
specialized consultants or prospective customers. Perhaps the taxpayer 
or special interest groups may also become involved. 

Some, in examining an instructional package are tempted to add another 
category, the So-what Analysis - "Now that you have gone to the work of 
preparing these materials, requirements, etc. - so what?" Comments such 
as these occur with sufficient frequency that many of us engage in one of 
three avoidance behaviors from time to time: 

1. We delay analysis as long as possible. 

2. If we can't avoid participating in the evaluation of a colleague's 
work, we superficially examine it with noncomittal praise. 

3. If we are playing the innovator role and receive a #2 response, we 
don't press the issue in order to obtain a more honest and exhaustive 
evaluation. 

Further exploration, amplification, and refinement of the basic concepts 
presented in this paper can assist in the avoidance of such pettiness. 
Hopefully, such development can help many would-be innovators through the 
crushing frustration of the inadequacies of one's first attempt, from which 
many don't rise. In addition, specification of criteria and accumulation of 
performance data of other similar instructional units and systems may stir- 
up undue complacency in those educators who need to be stirred up. 

Knowingly or unknowingly we somehow gather some data and evaluate 
the significance of what we are developing, and conclude that we are on 
the right track. Otherwise, we wouldn't exert the effort to develop what 
we do. Courageously utilizing increasingly effective evaluation procedures 
may not tell us exactly what we want to hear, but if we tune in carefully and 
listen carefully, we will hear some of what we should hear. 
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Solid x indicates no equivalent item on the other test 

Dashed x indicates partial overlap of the tasks of each of the tests. 
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